class: center, middle, inverse, title-slide .title[ # Calibrating Vulnerability Scores with Expert Opinions ] .subtitle[ ## When, Why and How! ] .author[ ### DIMA, Americas Bureau ] .date[ ### 10 November 2022 ] --- # What are vulnerability scores? Similarly to credit score in the private sector, vulnerability score offers a way to combine in a single number (i.e. a score for instance between 1 & 100) a composite measurement of vulnerability status in order to assess if a person can qualify for assistance --- # When do you need to use vulnerability scores? * Necessary once the population size exceed a certain scale – case by case management is not do-able or comes with high risk of exclusion * Minimize subjective component in assistance allocation * Measurement allow to prioritize assistance (Cash, but also potentially training, livelihood, micro-credit, etc.) in function of available budget --- # When not all eligible people can be prioritised.... .pull-left[ In protracted situations, not all eligible persons can be prioritised... ] .pull-right[  ] --- # When do you need to use expert opinions? .pull-left[ When we can not use simpler targeting approach - that does not require to distinguish eligibility from prioritization When we have experts.... ] .pull-right[  ] --- # When there's no representative dataset available When we have dataset that represents statistically well the whole population, it is possible to use statistical model, either: * proxy means testing: aka one outcome indicator can be used as single proxy of the vulnerability situation of the case (poverty, deprivation, etc.. )-> supervised classification * cluster population in consistent profiles that comes out from the data -> unsupervised classification - Item response Theory But in most humanitarian situation, achieving good statistical representativenesss is challenging (volatile environment, hidden population to sample..) --- # What are the challenges with expert opinions? * agree on the relevant eligibility criteria * agree on the relative importance of each criteria * agree on what other are agreeing... Without a specific facilitation approach, this can end into lengthy discussions... --- # Main challenges to create vulnerability scores Are selected indicators “add-able” in some sensical way, given the real-world meaning of the indicators? Can you add apples and oranges?  --- # The monkey dilemna: How to combine criteria? aka "compensability" for dummies  --- # Compensavility does not allow to reflect interactions between criteria  --- # An organised workshop facilitation 6 simple steps: * Step 1- Expert training * Step 2- Select the relevant eligibility criteria: quadratic voting * Step 3- Adjust the weight of criteria: conjoint analysis * Step 4- Review results and potentially iterate * Step 5- Implement the formula in the vulnerability scoring form * Step 6 - Apply Eligibility and prioritization threshold --- # Step 1- Expert training > "It is not enough to do your best; you must know what to do, and then do your best." > > W. Edwards Deming - Understand the difference between output variable and eligibility criteria --- # Step 2- Select the relevant eligibility criteria: quadratic voting  [Technical step by step tutorial: set up a quadratic vote on Kobotoolbox](quadraticVote.html) --- # Step 3- Adjust the weight of criteria: conjoint analysis Conjoint analysis is a type of consultation designed to measure the average opinion from a group of experts through specific pool where experts should compare different stereotypical profiles one by one and assess their respective vulnerability level. **Quotes:** > "If we have data, let’s look at data. If all we have are opinions, let’s go with mine!" > > Jim Barksdale [Technical step by step tutorial: set up a conjoint analysis on Kobotoolbox](conjointAnalysis.html) --- # Step 4- Review results and potentially iterate --- # Step 5- Implement the formula in the vulnerability scoring form Kobotoolbox form are often the default tool: The average weight obtained through conjoint analysis are implemented through a calculated filed using pow function Do not display the score during the screening - only provide information on eligibility --- # Step 6 - Eligible and prioritised Final scores are compared with 2 distinct threshold: - Eligibility threshold - applicants that should be covered based on their needs profile - Prioritization threshold - applicants that should be covered based on the available budget for the current assistance cohort --- # Conclusion When there's no representative data, expert opinion is the default options Creating buy-in on how to calibrate the vulnerability scoring formula is a key to the social acceptability Combining organised consultation (quadratic voting and conjoint analysis) allows to leverage collective intelligence As soon as you use a scoring system, the management of fluctuating prioritization threshold (i.e budget...) implies an efficient case management system: - Assistance cohort management - Continuous Vulnerability scoring - Appeal system